Siri: Okay, from now on I'll call you "an ambulance." Apple fixed this error shortly after its virtual assistant was first released in 2011. But a new contest shows that computers still lack the common sense required to avoid such embarrassing mix-ups. The results of the contest were presented at an academic conference in New York this week, and they provide some measure of how much work needs to be done to make computers truly intelligent. The Winograd Schema Challenge asks computers to make sense of sentences that are ambiguous but usually simple for humans to parse.

The Winograd Schema Challenge was proposed by Hector Levesque in 2011 as an alternative to the Turing Test. Chief among its features is a simple question format that can span many commonsense knowledge domains. Questions are chosen so that they do not require specialized knoweldge or training, and are easy for humans to answer. This article details our plans to run the WSC and evaluate results.